Introduction
Welcome to Unit 21, where we dive deep into the backward pass of neural networks, the heart of the training process.
Today's Focus:
- Backward Pass (Scalar): Understanding gradient computation for individual weights
- Walkthrough Example: Step-by-step calculation with concrete numbers
- Matrix Notation: Efficient computation for entire layers
- Python Implementation: Translating theory into code
This lecture builds upon your understanding of forward propagation and introduces the crucial backward propagation algorithm that enables neural networks to learn from data.
Theory
Recap: Cost Function
As in logistic regression, we define a loss/cost function that measures how far the network's output \(\hat{y}\) is from the true label \(y\).
Binary Cross-Entropy Loss (for a single example):
\[ L(y, \hat{y}) = -[y \cdot \ln(\hat{y}) + (1 - y) \cdot \ln(1 - \hat{y})] \]For m samples:
\[ L = -\frac{1}{m} \sum_{i=1}^{m} [y_i \cdot \ln(\hat{y}_i) + (1 - y_i) \cdot \ln(1 - \hat{y}_i)] \]Important: The output is now explicitly a function of all weights and biases in the hidden layer and output layer (and inputs):
\[
\hat{y} = \sigma \left(b_o + \sum_j w_{h_j,o} \sigma \left(b_{h_j} + \sum_i w_{i,h_j} x_i\right)\right)
\]
Key Insight: The cost function \(L\) depends on all the parameters (weights and biases) in the network. To train the network, we need to compute how much each parameter contributed to the error, which is what the backward pass does.
Example Dataset
Consider a small dataset of 6 observations with two features: Fat Score and Salt Score, and a binary outcome "Acceptance" (like/dislike).
| Obs. | Fat score | Salt score | Acceptance |
|---|---|---|---|
| 1 | 0.6 | 0.4 | like |
| 2 | 0.1 | 0.1 | dislike |
| 3 | 0.2 | 0.4 | dislike |
| 4 | 0.2 | 0.5 | dislike |
| 5 | 0.4 | 0.5 | like |
| 6 | 0.3 | 0.8 | like |
Network Architecture:
- Input layer: 2 features (Fat score, Salt score) \(\in \mathbb{R}^2\)
- Hidden layer: 3 units \(\in \mathbb{R}^3\)
- Output layer: 1 unit (probability of "like") \(\in \mathbb{R}^1\)
Note: If we remove the hidden layer (directly connect inputs to output with a sigmoid), we would essentially have logistic regression on Fat & Salt. The hidden layer lets us build a more expressive function.
Initialization
For our example, we use the first observation: \(x_1 = 0.6, x_2 = 0.4\)
Hidden Layer (3 neurons \(h_1, h_2, h_3\)):
- Neuron \(h_1\): \[ z_{h1} = b_{h1} + w_{1,h1}x_1 + w_{2,h1}x_2 \] \[ a_{h1} = \sigma(z_{h1}) \]
- Neuron \(h_2\): \[ z_{h2} = b_{h2} + w_{1,h2}x_1 + w_{2,h2}x_2 \] \[ a_{h2} = \sigma(z_{h2}) \]
- Neuron \(h_3\): \[ z_{h3} = b_{h3} + w_{1,h3}x_1 + w_{2,h3}x_2 \] \[ a_{h3} = \sigma(z_{h3}) \]
Initial weights and biases:
| Parameter | Value |
|---|---|
| Weights to \(h_1\): | \(w_{1,h1} = 0.5\), \(w_{2,h1} = -0.4\) |
| Bias \(b_{h1}\): | 0.1 |
| Weights to \(h_2\): | \(w_{1,h2} = -0.3\), \(w_{2,h2} = 0.8\) |
| Bias \(b_{h2}\): | -0.2 |
| Weights to \(h_3\): | \(w_{1,h3} = 0.2\), \(w_{2,h3} = 0.1\) |
| Bias \(b_{h3}\): | 0.05 |
| Weights from hidden to output: | \(w_{h1,o} = 0.7\), \(w_{h2,o} = -0.6\), \(w_{h3,o} = 0.3\) |
| Bias \(b_o\): | 0.2 |
Interactive Examples
Forward Pass
The forward pass computes the network's output step by step:
\[
x_1, x_2 \to [z_j = \sum w \cdot x + b] \to [a_j = \sigma(z_j)] \to [z_o = \sum w \cdot a + b] \to [\hat{y} = \sigma(z_o)] \to L
\]
Hidden Layer Computations:
\[
\begin{array}{l}
z_{h1} = b_{h1} + w_{1,h1}x_1 + w_{2,h1}x_2 = 0.1 + (0.5)(0.6) + (-0.4)(0.4) = 0.1 + 0.30 - 0.16 = 0.24 \\
a_{h1} = \sigma(0.24) \approx 0.5597
\end{array}
\]
\[
\begin{array}{l}
z_{h2} = b_{h2} + w_{1,h2}x_1 + w_{2,h2}x_2 = -0.2 + (-0.3)(0.6) + (0.8)(0.4) = -0.2 - 0.18 + 0.32 = -0.06 \\
a_{h2} = \sigma(-0.06) \approx 0.4850
\end{array}
\]
\[
\begin{array}{l}
z_{h3} = b_{h3} + w_{1,h3}x_1 + w_{2,h3}x_2 = 0.05 + (0.2)(0.6) + (0.1)(0.4) = 0.05 + 0.12 + 0.04 = 0.21 \\
a_{h3} = \sigma(0.21) \approx 0.5523
\end{array}
\]
Output Layer Computation:
\[
\begin{array}{l}
z_o = b_o + w_{h1,o}a_{h1} + w_{h2,o}a_{h2} + w_{h3,o}a_{h3} \\
= 0.2 + (0.7)(0.5597) + (-0.6)(0.4850) + (0.3)(0.5523) \\
= 0.2 + 0.3918 - 0.2910 + 0.1657 = 0.4665
\end{array}
\]
\[
\hat{y} = \sigma(0.4665) \approx 0.6147
\]
Result: The network's output \(\hat{y} \approx 0.6147\), which could be interpreted as the probability of "like".
Backward Pass: Update Rules
During training, we update the parameters using gradient descent:
- Output Bias: \(b_o \gets b_o - \alpha \frac{\partial L}{\partial b_o}\)
- Weights from hidden to output: \(w_{h_j,o} \gets w_{h_j,o} - \alpha \frac{\partial L}{\partial w_{h_j,o}}\) for each j
- Hidden biases: \(b_{h_j} \gets b_{h_j} - \alpha \frac{\partial L}{\partial b_{h_j}}\) for each j
- Weights from Input to Hidden: \(w_{i,h_j} \gets w_{i,h_j} - \alpha \frac{\partial L}{\partial w_{i,h_j}}\) for each i, j
Important Note: Use the original weight values from the forward pass, not updated weights. All gradients must be computed using the current network state before any updates occur.
Theory: Backward Pass Equations
Step 1: Output Layer Gradient
Gradient for output bias:
\[ \frac{\partial L}{\partial b_o} = \hat{y} - y \]Gradient for weights from hidden to output:
\[ \frac{\partial L}{\partial w_{h_j,o}} = (\hat{y} - y) \cdot a_j \quad \text{for each } j \]Where:
- \(a_j\): activation of hidden neuron \(h_j\)
- \(w_{h_j,o}\): weight from hidden neuron j to output
Step 2: Hidden Layer Gradients
2a: Error at hidden layer activations:
\[ \frac{\partial L}{\partial a_j} = (\hat{y} - y) \cdot w_{h_j,o} \]2b: Error at hidden pre-activation (chain rule):
\[ \frac{\partial L}{\partial z_j} = \frac{\partial L}{\partial a_j} \cdot \sigma'(z_j) = \frac{\partial L}{\partial a_j} \cdot a_j \cdot (1 - a_j) \]2c: Gradient for hidden bias:
\[ \frac{\partial L}{\partial b_{h_j}} = \frac{\partial L}{\partial z_j} \]2d: Gradient for hidden weights:
\[ \frac{\partial L}{\partial w_{i,h_j}} = \frac{\partial L}{\partial z_j} \cdot x_i \quad \text{for each input } i \]Numerical Solutions
Backward Pass Working Example
Given: \(\hat{y} \approx 0.6147\), true label \(y = 1\), learning rate \(\eta = 0.1\)
Step 1: Compute error term at output
\[
\delta_o = \hat{y} - y = 0.6147 - 1 = -0.3853
\]
(Note: this arises from \(\frac{\partial L}{\partial z_o}\) given cross-entropy + sigmoid.)
Step 2: Update output weight \(w_{h1,o} = 0.7\) (initial)
\[
\frac{\partial L}{\partial w_{h1,o}} = \delta_o \cdot a_{h1} = (-0.3853) \cdot (0.5597) \approx -0.2156
\]
Weight update:
\[ w_{h1,o}^{\text{new}} = w_{h1,o}^{\text{old}} - \eta \times \frac{\partial L}{\partial w_{h1,o}} = 0.7 - 0.1 \times (-0.2156) = 0.7 + 0.02156 = 0.72156 \]Update bias \(b_o\):
\[ \frac{\partial L}{\partial b_o} = \delta_o \cdot 1 = -0.3853 \] \[ b_o^{\text{new}} = 0.2 - 0.1 \times (-0.3853) = 0.2 + 0.03853 = 0.23853 \]Step 3: Back-propagate to hidden layer
Compute hidden neuron \(h_2\) terms:
\(a_{h2} \approx 0.4850, \quad z_{h2} \approx -0.06\)
Compute hidden neuron error term:
\[ \delta_{h2} = \delta_o \cdot w_{h2,o} \cdot a_{h2} (1 - a_{h2}) = (-0.3853) \cdot (-0.6) \cdot (0.4850 \times 0.5150) \] \[ = 0.23118 \cdot (0.2498) \approx 0.0577 \]Gradient for \(w_{1,h2}\) (input \(x_1 = 0.6\)):
\[ \frac{\partial L}{\partial w_{1,h2}} = \delta_{h2} \cdot x_1 = 0.0577 \cdot 0.6 \approx 0.03462 \]Update \(w_{1,h2}\):
\[ w_{1,h2}^{\text{new}} = -0.3 - 0.1 \times (0.03462) = -0.3 - 0.003462 = -0.303462 \]Update bias of \(h_2\):
\[ \frac{\partial L}{\partial b_{h2}} = \delta_{h2} \cdot 1 = 0.0577 \] \[ b_{h2}^{\text{new}} = -0.2 - 0.1 \times 0.0577 = -0.2 - 0.00577 = -0.20577 \]Theory: Forward and Backward Pass in Matrix Notation
Forward Pass (Matrix Notation)
Hidden Layer:
\[ z^{[1]} = (W^{[1]})^T x + b^{[1]} \] \[ a^{[1]} = \sigma(z^{[1]}) \]Output Layer:
\[ z^{[2]} = (W^{[2]})^T a^{[1]} + b^{[2]} \] \[ \hat{y} = \sigma(z^{[2]}) \]Dimensions:
\[
\begin{array}{l}
W^{[1]} \in \mathbb{R}^{2 \times 3}, \quad x \in \mathbb{R}^{2 \times 1}, \quad b^{[1]} \in \mathbb{R}^{3 \times 1} \\
W^{[2]} \in \mathbb{R}^{3 \times 1}, \quad b^{[2]} \in \mathbb{R}^{1 \times 1}
\end{array}
\]
Backward Pass (Matrix Notation)
Output Layer Gradients:
\[ \frac{\partial L}{\partial b^{[2]}} = \hat{y} - y \] \[ \frac{\partial L}{\partial W^{[2]}} = (\hat{y} - y) (a^{[1]})^T \]Hidden Layer Gradients:
\[ \frac{\partial L}{\partial b^{[1]}} = W^{[2]} (\hat{y} - y) \odot a^{[1]} \odot (1 - a^{[1]}) \] \[ \frac{\partial L}{\partial W^{[1]}} = \left[ W^{[2]} (\hat{y} - y) \odot a^{[1]} \odot (1 - a^{[1]}) \right] x^T \]Where \(\odot\) represents element-wise multiplication.
Key Insight: Matrix notation allows us to compute all gradients for a layer in a single operation, rather than computing each weight's gradient individually. This is much more efficient, especially for large networks.
Python Implementation
Helper Functions
def sigmoid(z):
return 1.0 / (1 + np.exp(-z))
def sigmoid_derivative(z):
return sigmoid(z) * (1.0 - sigmoid(z))
def predict(x, model):
W1 = model['W1']
b1 = model['b1']
W2 = model['W2']
b2 = model['b2']
Z1 = np.matmul(W1, x) + b1
A1 = sigmoid(Z1)
Z2 = np.matmul(W2, A1) + b2
A2 = sigmoid(Z2)
return A2
Training Function
def train(X, y, m_hidden, learning_rate, n_iter):
n_input, m = X.shape
# Initialize weights
W1 = np.random.randn(m_hidden, n_input)
b1 = np.zeros((m_hidden, 1))
W2 = np.random.randn(1, m_hidden)
b2 = np.zeros((1, 1))
for i in range(1, n_iter + 1):
# ==== Forward pass ====
Z1 = np.matmul(W1, X) + b1
A1 = sigmoid(Z1)
Z2 = np.matmul(W2, A1) + b2
A2 = sigmoid(Z2) # y_hat
# ==== Backward pass ====
dZ2 = A2 - y
dW2 = np.matmul(dZ2, A1.T) / m
db2 = np.sum(dZ2, axis=1, keepdims=True) / m
dA1 = np.matmul(W2.T, dZ2)
dZ1 = dA1 * sigmoid_derivative(Z1)
dW1 = np.matmul(dZ1, X.T) / m
db1 = np.sum(dZ1, axis=1, keepdims=True) / m
# ==== Parameter update ====
W1 -= learning_rate * dW1
b1 -= learning_rate * db1
W2 -= learning_rate * dW2
b2 -= learning_rate * db2
# Optional training log
if i % 100 == 0:
cost = np.mean((y - A2)**2)
print(f"Iteration {i}, training loss: {cost}")
return {'W1': W1, 'b1': b1, 'W2': W2, 'b2': b2}
Key Implementation Notes:
- Vectorized operations: The implementation uses matrix operations (np.matmul) for efficiency
- Sigmoid derivative: Uses the property that \(\sigma'(z) = \sigma(z) \cdot (1 - \sigma(z))\)
- Gradient computation: Follows the chain rule to compute gradients for each layer
- Weight updates: Uses gradient descent with the specified learning rate
- Training log: Optionally prints the loss every 100 iterations
Try It Yourself
Problem 1: Forward Pass Calculation
Given the following network parameters and input:
- Input: \(x_1 = 0.5, x_2 = 0.3\)
- Hidden neuron \(h_1\): \(w_{1,h1} = 0.4, w_{2,h1} = -0.2, b_{h1} = 0.1\)
- Output neuron: \(w_{h1,o} = 0.6, b_o = 0.2\)
Tasks:
- Calculate \(z_{h1}\) and \(a_{h1}\)
- Calculate \(z_o\) and \(\hat{y}\)
Solution:
- Hidden neuron calculation: \[ z_{h1} = 0.1 + (0.4)(0.5) + (-0.2)(0.3) = 0.1 + 0.2 - 0.06 = 0.24 \] \[ a_{h1} = \sigma(0.24) \approx 0.5597 \]
- Output neuron calculation: \[ z_o = 0.2 + (0.6)(0.5597) = 0.2 + 0.3358 = 0.5358 \] \[ \hat{y} = \sigma(0.5358) \approx 0.6306 \]
Problem 2: Backward Pass Calculation
Using the network from Problem 1, assume:
- True label \(y = 1\)
- Learning rate \(\eta = 0.1\)
- Predicted output \(\hat{y} \approx 0.6306\)
Tasks:
- Calculate the error term \(\delta_o\)
- Calculate the gradient \(\frac{\partial L}{\partial w_{h1,o}}\)
- Update the weight \(w_{h1,o}\)
- Update the bias \(b_o\)
Solution:
- Error term: \[ \delta_o = \hat{y} - y = 0.6306 - 1 = -0.3694 \]
- Gradient for \(w_{h1,o}\): \[ \frac{\partial L}{\partial w_{h1,o}} = \delta_o \cdot a_{h1} = (-0.3694) \cdot (0.5597) \approx -0.2075 \]
- Update \(w_{h1,o}\): \[ w_{h1,o}^{\text{new}} = 0.6 - 0.1 \times (-0.2075) = 0.6 + 0.02075 = 0.62075 \]
- Update \(b_o\): \[ \frac{\partial L}{\partial b_o} = \delta_o = -0.3694 \] \[ b_o^{\text{new}} = 0.2 - 0.1 \times (-0.3694) = 0.2 + 0.03694 = 0.23694 \]
Problem 3: Matrix Notation Forward Pass
Given:
- Input vector \(x = \begin{bmatrix} 0.5 \\ 0.3 \end{bmatrix}\)
- Weight matrix \(W^{[1]} = \begin{bmatrix} 0.4 & -0.2 \\ -0.1 & 0.3 \\ 0.2 & 0.1 \end{bmatrix}\)
- Bias vector \(b^{[1]} = \begin{bmatrix} 0.1 \\ -0.1 \\ 0.05 \end{bmatrix}\)
Task: Calculate \(z^{[1]}\) and \(a^{[1]}\) using matrix notation.
Solution:
Calculate \(z^{[1]}\):
\[
z^{[1]} = (W^{[1]})^T x + b^{[1]} = \begin{bmatrix} 0.4 & -0.1 & 0.2 \\ -0.2 & 0.3 & 0.1 \end{bmatrix} \begin{bmatrix} 0.5 \\ 0.3 \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.1 \\ 0.05 \end{bmatrix}
\]
\[
= \begin{bmatrix} (0.4)(0.5) + (-0.1)(0.3) + (0.2)(?) \\ (-0.2)(0.5) + (0.3)(0.3) + (0.1)(?) \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.1 \\ 0.05 \end{bmatrix}
\]
Note: There seems to be a dimension mismatch. \(W^{[1]}\) should be \(2 \times 3\) (input features × hidden units), so \((W^{[1]})^T\) is \(3 \times 2\), and \(x\) is \(2 \times 1\), giving \(z^{[1]}\) as \(3 \times 1\).
\[ z^{[1]} = \begin{bmatrix} 0.4 & -0.2 \\ -0.1 & 0.3 \\ 0.2 & 0.1 \end{bmatrix}^T \begin{bmatrix} 0.5 \\ 0.3 \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.1 \\ 0.05 \end{bmatrix} \] \[ = \begin{bmatrix} 0.4 & -0.1 & 0.2 \\ -0.2 & 0.3 & 0.1 \end{bmatrix} \begin{bmatrix} 0.5 \\ 0.3 \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.1 \\ 0.05 \end{bmatrix} \] \[ = \begin{bmatrix} (0.4)(0.5) + (-0.1)(0.3) \\ (-0.2)(0.5) + (0.3)(0.3) \\ (0.2)(0.5) + (0.1)(0.3) \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.1 \\ 0.05 \end{bmatrix} \] \[ = \begin{bmatrix} 0.2 - 0.03 \\ -0.1 + 0.09 \\ 0.1 + 0.03 \end{bmatrix} + \begin{bmatrix} 0.1 \\ -0.1 \\ 0.05 \end{bmatrix} = \begin{bmatrix} 0.28 \\ -0.02 \\ 0.18 \end{bmatrix} \]Calculate \(a^{[1]}\): Apply sigmoid to each element of \(z^{[1]}\)
\[
a^{[1]} = \sigma(z^{[1]}) = \begin{bmatrix} \sigma(0.28) \\ \sigma(-0.02) \\ \sigma(0.18) \end{bmatrix} \approx \begin{bmatrix} 0.5698 \\ 0.4950 \\ 0.5445 \end{bmatrix}
\]
Problem 4: Backward Pass Matrix Notation
Given:
- Output \(\hat{y} = 0.7\), true label \(y = 1\)
- Hidden layer activations \(a^{[1]} = \begin{bmatrix} 0.6 \\ 0.4 \\ 0.5 \end{bmatrix}\)
- Output weights \(W^{[2]} = \begin{bmatrix} 0.5 & -0.3 & 0.2 \end{bmatrix}\)
Tasks:
- Calculate \(\frac{\partial L}{\partial b^{[2]}}\)
- Calculate \(\frac{\partial L}{\partial W^{[2]}}\)
Solution:
- Output bias gradient: \[ \frac{\partial L}{\partial b^{[2]}} = \hat{y} - y = 0.7 - 1 = -0.3 \]
- Output weight gradient: \[ \frac{\partial L}{\partial W^{[2]}} = (\hat{y} - y) (a^{[1]})^T = (-0.3) \begin{bmatrix} 0.6 & 0.4 & 0.5 \end{bmatrix} = \begin{bmatrix} -0.18 & -0.12 & -0.15 \end{bmatrix} \]
Problem 5: Chain Rule Application
Consider a simple network with:
- Input \(x\)
- Hidden layer: \(z_h = w_1 x + b_1\), \(a_h = \sigma(z_h)\)
- Output layer: \(z_o = w_2 a_h + b_2\), \(\hat{y} = \sigma(z_o)\)
- Loss: \(L = (\hat{y} - y)^2\)
Task: Derive the expression for \(\frac{\partial L}{\partial w_1}\) using the chain rule.
Solution:
Using the chain rule:
\[
\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z_o} \cdot \frac{\partial z_o}{\partial a_h} \cdot \frac{\partial a_h}{\partial z_h} \cdot \frac{\partial z_h}{\partial w_1}
\]
Compute each term:
- \(\frac{\partial L}{\partial \hat{y}} = 2(\hat{y} - y)\)
- \(\frac{\partial \hat{y}}{\partial z_o} = \sigma'(z_o) = \hat{y}(1 - \hat{y})\)
- \(\frac{\partial z_o}{\partial a_h} = w_2\)
- \(\frac{\partial a_h}{\partial z_h} = \sigma'(z_h) = a_h(1 - a_h)\)
- \(\frac{\partial z_h}{\partial w_1} = x\)
Combine:
\[
\frac{\partial L}{\partial w_1} = 2(\hat{y} - y) \cdot \hat{y}(1 - \hat{y}) \cdot w_2 \cdot a_h(1 - a_h) \cdot x
\]
Interactive Quiz
Test your understanding of the Backward Pass in Neural Networks:
Question 1: What is the primary purpose of the backward pass?
Question 2: In the backward pass, which of the following is computed first?
Question 3: What is the derivative of the sigmoid function \(\sigma(z)\)?
Question 4: In matrix notation, what is the dimension of \(W^{[1]}\) for a network with 2 input features and 3 hidden units?
Question 5: Why do we use the original weight values (not updated) when computing all gradients in a single backward pass?
Key Takeaways
Backward Pass Fundamentals:
- Purpose: Computes gradients of the loss function with respect to all parameters (weights and biases)
- Mechanism: Uses the chain rule to propagate errors backward through the network
- Order: Compute output layer gradients first, then hidden layer gradients
- Synchronous updates: All gradients must be computed using the same network state (original weights) before any updates occur
Output Layer Gradients:
- Bias gradient: \(\frac{\partial L}{\partial b_o} = \hat{y} - y\)
- Weight gradient: \(\frac{\partial L}{\partial w_{h_j,o}} = (\hat{y} - y) \cdot a_j\) for each hidden unit j
- Error term: \(\delta_o = \hat{y} - y\) (for cross-entropy + sigmoid)
Hidden Layer Gradients:
- Activation error: \(\frac{\partial L}{\partial a_j} = (\hat{y} - y) \cdot w_{h_j,o}\)
- Pre-activation error: \(\frac{\partial L}{\partial z_j} = \frac{\partial L}{\partial a_j} \cdot \sigma'(z_j) = \frac{\partial L}{\partial a_j} \cdot a_j \cdot (1 - a_j)\)
- Bias gradient: \(\frac{\partial L}{\partial b_{h_j}} = \frac{\partial L}{\partial z_j}\)
- Weight gradient: \(\frac{\partial L}{\partial w_{i,h_j}} = \frac{\partial L}{\partial z_j} \cdot x_i\)
Matrix Notation:
- Efficiency: Allows computing all gradients for a layer in a single operation
- Forward pass: \(z^{[l]} = (W^{[l]})^T a^{[l-1]} + b^{[l]}\), \(a^{[l]} = \sigma(z^{[l]})\)
- Backward pass: Uses element-wise operations (\(\odot\)) for efficient computation
Python Implementation:
- Vectorized: Use matrix operations for efficiency
- Sigmoid derivative: \(\sigma'(z) = \sigma(z) \cdot (1 - \sigma(z))\)
- Chain rule: Follow the chain rule to compute gradients layer by layer
- Weight updates: \(w \gets w - \eta \cdot \frac{\partial L}{\partial w}\)
Common Pitfalls
⚠️ Backward Pass:
- Using updated weights: Computing gradients with updated weights instead of original weights can lead to incorrect gradient estimates
- Incorrect chain rule application: Missing terms in the chain rule can lead to wrong gradients
- Order of operations: Computing hidden layer gradients before output layer gradients breaks the dependency chain
- Numerical instability: Very small or very large gradients can cause numerical issues
- Vanishing gradients: For deep networks, gradients can become extremely small, causing early layers to learn very slowly
⚠️ Matrix Notation:
- Dimension mismatch: Incorrect matrix dimensions can cause errors in gradient computation
- Transpose errors: Forgetting to transpose weight matrices can lead to incorrect dimensions
- Element-wise vs matrix multiplication: Confusing \(\odot\) (element-wise) with matrix multiplication
- Broadcasting issues: In NumPy, broadcasting rules might lead to unexpected behavior if dimensions aren't carefully managed
⚠️ Implementation:
- Learning rate too large: Can cause weights to oscillate or diverge
- Learning rate too small: Can lead to very slow convergence
- Not initializing weights: Starting with all zeros can prevent the network from learning
- Numerical precision: Using float32 instead of float64 can cause precision issues in some cases
- Debugging: It can be challenging to debug backward pass implementations; unit tests are essential
⚠️ Conceptual:
- Confusing forward and backward: The forward pass computes outputs, the backward pass computes gradients
- Local vs global minima: Gradient descent finds local minima, not necessarily global minima
- Convexity: Neural network loss functions are typically non-convex, having many local minima
Resources
📚 Neural Networks:
- Deep Learning Book by Goodfellow, Bengio, and Courville - Comprehensive resource
- CS231n: Convolutional Neural Networks for Visual Recognition - Stanford course notes
- TensorFlow Playground - Interactive neural network visualization
📚 Backpropagation:
- Backpropagation Explained by Christopher Olah - Excellent visual explanation
- A Step-by-Step Backpropagation Example - Detailed walkthrough
- Backpropagation in Neural Networks - Video tutorial
📚 Mathematical Foundations:
- Khan Academy: Multivariable Calculus - Chain rule, partial derivatives
- Matrix Calculus by 3Blue1Brown
📖 Books:
- Machine Learning with PyTorch and Scikit-Learn by Raschka et al.
- An Introduction to Statistical Learning by James et al.
- Pattern Recognition and Machine Learning by Bishop
💻 Practical Implementation:
- NumPy Documentation - Essential for matrix operations
- Google Colab - Free environment to experiment with neural networks
- Neural Networks from Scratch - Practical implementation guide